Weak valued statistics as fundamental explanation of quantum physics 
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Recently, weak measurements have attracted a lot of interest as an experimental method for 
the investigation of non-classical correlations between observables that cannot be measured jointly. 
Here, I explain how the complex valued statistics observed in weak measurements relate to the 
operator algebra of the conventional Hilbert space formalism and show that the algebra of operators 
originates from more fundamental relations between the physical properties of a quantum system 
that can be expressed in terms of complex conditional probabilities. In particular, commutation 
relations can be identified with fundamental imaginary correlations that characterize the relations 
between physical properties in terms of their transformation dynamics. Non-commutativity thus 
originates from a definition of relations between physical properties that replaces the assumption of 
joint reality with a complex-valued probability reflecting the dynamical response of the system to 
external forces, e.g. in measurement interactions. 



I. INTRODUCTION 

In the most general formulation of quantum mechan- 
ics, classical variables are replaced by Hilbert space oper- 
ators acting on state vectors. If one of these observables 
is known, the system is considered to be in an eigenstate 
of the operator, which can be written as a superposition 
of any other set of eigenvectors. This is the point where 
the physical meaning is unclear. In classical physics, a 
system is described by simultaneously assigning values 
to all its properties. One might expect that such an as- 
signment should also be possible in quantum mechanics, 
since an appropriate measurement will always result in a 
single precise outcome. Why is it not possible to replace 
the probabilities of measurement outcomes predicted by a 
quantum state by the actual values of the physical prop- 
erties observed in the measurement? Clearly, the fact 
that operators cannot just be replaced by their eigenval- 
ues indicates a very fundamental difference between the 
physics described by operators and the physics of classi- 
cal variables. A proper explanation of quantum physics 
should identify this difference by explaining in detail how 
the various properties of a quantum system are related 
to each other, and how the results obtained in different 
measurements can be explained in terms of these fun- 
damental relations. Unfortunately, the original formula- 
tion of operator algebra was not based on a general the- 
ory of measurements, but established the mathematical 
structure in terms of approximate analogies such as the 
wave-particle dualism, which are not sufficiently general 
to address all of the questions raised by a thorough in- 
vestigation of quantum measurements. The consequence 
has been a string of seemingly paradoxical results that 
can be predicted and described using the standard the- 
ory, but appear to have no satisfactory explanation in 
terms of the underlying physics. 
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It may seem strange that predictions of a widely ac- 
cepted formalism have no physical explanation. However, 
quantum theory was constructed from very little empir- 
ical evidence, and it was not initially clear which details 
of the theory would ever be accessible to experimental 
tests. Based on this lack of evidence, the terminology 
was developed by analogies with classical concepts, even 
where these concepts directly contradicted the relations 
between their quantum mechanical analogs. This is most 
clearly seen in the operator formalism, e.g. when the spin 
is defined as a three dimensional vector, even though it is 
never possible to identify a direction based on a simulta- 
neous assignment of three eigenvalues to the three non- 
commuting components. Curiously, the assignment of 
non-commuting operators to the three orthogonal com- 
ponents does reproduce the transformations associated 
with spatial rotations, and this seems to correspond to 
the relations between averages observed in measurements 
along different spatial directions. It is therefore possible 
to reduce the problem to a more specific question: how 
are the individual measurement outcomes represented by 
eigenstates related to the algebra of operators that re- 
produces the transformations between different "orienta- 
tions," or contexts, of a quantum measurement? 

If one recognizes that this is in fact an open question, 
it is possible to get new insights from the results recently 
reported in the context of weak measurements. In partic- 
ular, weak measurements can directly establish a relation 
between three non-commuting properties of a quantum 
system by combining preparation and a precise final mea- 
surement with the intermediate measurement of a third 
property. Quantum states can then be characterized by 
complex joint probabilities of only two non-commuting 
properties, as long as the eigenstates of the two prop- 
erties have non-zero mutual overlap [3-01 • Weak mea- 
surements thus provide a better understanding of quan- 
tum states in terms of fundamental statistics. However, 
the essential point might actually be that weak measure- 
ments can actually reveal fundamental laws of physics 
that do not depend on the specific quantum state and are 
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universally valid in any context. As pointed out in [H, [5|, 
the complex conditional probabilities observed in weak 
measurements describe the fundamental relation of three 
physical properties in terms of transformations between 
eigenstates of two properties generated by the third. In 
the following, I intend to argue that these relations are 
the natural explanation of the operator algebra, and that 
the conventional operator formalism is best understood 
as a theory of complex conditional probabilities. 



II. OPERATOR PHYSICS AND 
UNCERTAINTIES 

The key observation that makes new insights into the 
quantum formalism possible is that the original Hilbert 
space formalism is more complete and more precise than 
the textbook examples suggest. In particular, it is pos- 
sible to analyze the precision of statements about unob- 
served properties, as explained by Ozawa in his ground- 
breaking work on measurement uncertainties @ . Specif- 
ically, Ozawa introduced a mathematical expression for 
the error of a measurement when neither the initial state 
nor the measurement results are represented by an eigen- 
state of the target observable. The very fact that such an 
expression exists proves that there is more to the operator 
formalism than the textbooks would tell. Interestingly, 
it was pointed out soon after by Hall that Ozawa's un- 
certainties are minimized by the real parts of the weak 
values Q. Indeed, the proper conclusion seems to be that 
the results of weak measurements represent the zero un- 
certainty values of an operator for a specific selection of 
initial pure state and precise final measurement 
Specifically the uncertainty of A in an initial state | ip) 
can be fully explained by the fluctuations of the complex 
weak values observed in any final measurement with out- 
comes I m): 
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Since the weak values conditioned by | ijj) and | m) ap- 
pear to provide a complete description of the quantum 
fluctuations of A in | ip), it is possible to think of the op- 
erator A as a "function" determined by the initial state 
and the final state, even though the values of this func- 
tion do not correspond to the eigenvalues of A. In fact, 
this functional dependence of the value of A on a pair 
of non-commuting eigenstates was already introduced by 
Dirac as early as 1945 [ll|, who thus discovered weak 
values 43 years before the actual introduction of weak 
measurements. However, there is a problem with this 
representation of an operator A as a function of two pre- 
cise measurement outcomes. If the weak value were the 
true value of A for a reality jointly defined by | ip) and 
m), we would expect to find that the weak value of A 2 
would be equal to the square of the weak value. The op- 
erator formalism shows that this is not the case. Instead, 



weak values seem to have complex-valued uncertainties 
that average to zero in Eq. . 

The failure of a naive realist interpretation of weak 
values suggests a return to eigenvalues as an alternative. 
Using the spectral decomposition of the operator A, the 
weak value can be written as an average defined by com- 
plex conditional probabilities, 
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A a p(a\ip,m) 



(2) 



where p(a\ip,m) is given by the complex weak value of 
the projector | a) (a |. Therefore, weak values are con- 
sistent with the eigenvalues A a if they are interpreted as 
an average of complex conditional probabilities that de- 
scribe the relation between the measurement results ip, 
m, and a. Significantly, it is not necessary to interpret 
these weak values as specific measurement results. In- 
stead, they are elements of a measurement independent 
representation of A [j| . It is therefore possible to express 
all mathematical relations between operators in terms 
of complex conditional probabilities, independent of the 
specific measurement context. 

Since the non-commutativity of operators is often em- 
phasized as a key difference between quantum physics 
and classical physics, it may be worth taking a closer 
look at the complex statistics expressed by commuta- 
tion relations. In particular, the uncertainty limit of two 
non-commuting observables for pure states | ip) is often 
explained in terms of the expectation value of the com- 
mutation relation, 



AAAB > -\(ip | [A,B] | ip)\ 



(3) 



Using the spectral decomposition of the operators A and 
B, the commutation relation can be written as a sum 
over all possible combinations of measurement outcomes 
a and 6, with a complex joint probability of p(a, b\ip) 
obtained for the quantum state | ip). The expectation 
value of the commutation relation then corresponds to 
the imaginary part of the complex correlation between A 
and B in p(et, b\ip), and the uncertainty limit of Eq.Q is 
given by 

l -{ip I [A, B] | V) = E A « B > Im M a > • ( 4 ) 

a, b 

Non-commutativity can therefore be explained in terms 
of complex joint probabilities. The standard uncertainty 
limit given by Eq.Q is then a natural consequence of 
the classical rule that correlations cannot be larger than 
the product of the fluctuations, which applies equally to 
complex and to real valued probabilities. 

It is possible to measure the complete complex proba- 
bility p(a,b\ip), e.g. by performing a weak measurement 
of the projector on | a) followed by a final measurement of 
b Q . The joint probability is then given by the product 
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of weak value and measurement probability, 

p(a,b\i/;)=p(a\i,,b)\(b\^}\ 2 

= (b\a){a\^)^\b). (5) 

Importantly, this joint probability expresses a highly 
symmetric relation between the three properties repre- 
sented by a, 6, and ip. If we replace | tp) with the eigen- 
states | m) of a third property M, the relations between 
A, B and M are determined by a set of d 3 joint probabil- 
ities that represent any of the three eigenstates in terms 
of joint probabilities of the two other eigenstates, 

p(a,b\m) = p(m,a\b) = p(b,m\a). (6) 

Therefore, the initial state in the complex joint proba- 
bility for a system with know property to is fully deter- 
mined by a fundamental relation between to, a and 6, 
even though future measurements will reveal additional 
information on a and/or b. Specifically, the uncertainty 
free relations p(a\m, b) require that any initial knowledge 
of the value of m restricts the possible knowledge of b, 
so that all predictable probabilities will be positive and 
real. The uncertainty expressed by the probability distri- 
bution |(6 | to) | 2 is therefore a necessary consequence of 
the positivity of predicted probabilities and does not rep- 
resent an arbitrary reduction of the correlations between 
to and b. In quantum mechanics, the complex joint prob- 
abilities p(a, b\m) of pure states | m) are therefore fully 
determined by fundamental laws of physics, despite the 
fact that the individual measurement outcomes of either 
a or b are unpredictable and random. 



III. QUANTUM DETERMINISM 

Weak measurement overcomes the uncertainty limit 
by combining initial information with final information. 
The operator formalism suggests that the weak values so 
obtained are indeed uncertainty free, but that the con- 
ditional probabilities for the different outcomes of pro- 
jective weak measurements are complex. Since complex 
probabilities cannot be interpreted as relative frequen- 
cies, this result suggests that the realities of (a, b) are 
fundamentally different from the realities of (to, a) and 
from the realities of (6, to) . The meaning of complex joint 
probabilities must therefore be found elsewhere - specifi- 
cally, in the relation between different possible measure- 
ments that can never be performed on the same system. 
Thus, the question is not a counter factual one (what 
would have happened in this specific system if we had 
measured to instead of &?), but a statistical one (how to 
explain the statistics of m-measurements in terms of the 
statistics obtained in unrelated measurements of a and b) . 
Complex conditional probabilities are fundamental be- 
cause they allow a reliable prediction of probabilities for 
measurement outcomes to from the complex joint proba- 
bilities of the measurement outcomes a and b observed in 



separate measurements on the same kind of system [5|, 
p(m) =^2p(m\a,b)p(a,b\ip). (7) 

a.b 

Here, the conditional probability p(m\a, b) is a univer- 
sal relation between a, 6, and to that does not depend 
on the initial state tf> or the measurements used to de- 
termine p(a,b\tp). No matter what the specific situation 
describe by the initial state i\) is, these conditions can 
always be characterized by an experimentally observable 
complex joint probability of a and b, and the relation be- 
tween this joint probability and any other measurement 
to is fundamentally defined by p(m\a, b). Significantly, 
the complex conditional probability p(m\a, b) is the only 
relation that quantum physics allows between A, B, and 
M . The fundamental nature of this relation can be il- 
lustrated by expressing the operator M in terms of the 
weak value statistics for a and 6, 

M = ^M m | m)(m 

m 

p(m\a, b) | b)(b | a)(a | . (8) 

a,b,m 

The conditional probabilities p(m\a, b) therefore describe 
the relation between the operator M and the operators 
A and B in terms of the spectral decompositions of the 
operators A and B given by their eigenstates {| a)} and 
{| b)}. Based on this relation, the complete algebra of 
Hilbert space operators can be derived from the complex 
conditional probabilities. 

Since they are based on eigenstates and precise mea- 
surement results, complex conditional probabilities can 
provide a more detailed explanation of operator relations, 
where the non-classical properties often average out in 
the sum over all possible eigenstates and eigenvalues. In 
particular, the important analogies between the operator 
formalism and classical physics are mostly meaningful 
when the statistics is described in terms of expectation 
values. However, this is an approximation that is only 
valid when the effects of the necessary uncertainties are 
small enough to be neglected. It is therefore wrong to 
interpret the case where classical relations hold as ab- 
sence of quantum correlations. Even in the most classical 
case, the microscopic relations between exact properties 
are given by complex probabilities of the form p{m\a, b), 
which do not depend on the available information or the 
quantum state under consideration, and do not change 
into a more classical form as the uncertainties of the ini- 
tial state increase. Therefore, it is always a mistake to 
assume that physical properties describe a separate ma- 
terial reality independent of interactions. Instead, even 
the classical limit should be understood as a reality of 
interactions and effects. It may be useful to remember 
how Dr. Johnson tried to refute Berkeley (and failed): 
the existence of a stone is known by kicking it, not by a 
dogmatic belief in the fundamental reality of its geomet- 
ric shape in space. Interaction independent existence is 
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a misinterpretation of reality, since reality is known by 
touch and sight. In the classical limit, the effects of in- 
teractions may be conveniently summarized by assigning 
approximate values to all properties, but it is important 
to keep in mind that the exact values of the properties 
a, b and m only refer to potential experiences of the ob- 
ject, which emerge gradually in the interaction with the 
object. In quantum physics, the effects of these interac- 
tion processes cannot be separated from reality, and the 
relations between different possible observations need to 
include the complex phases associated with the action of 
transformations. 



IV. TRANSFORMATION DYNAMICS 

Traditionally, much of physics is concerned with dy- 
namics, and therefore with the changes of a specific prop- 
erty in time. However, it is important to recognize that 
quantum mechanics provides a description of time evo- 
lution that is actually incompatible with a description 
of dynamics as a continuous evolution of reality. The 
proper description of time evolution is given in terms of 
unitary operators that may be applied cither to trans- 
form the states (Schroedinger picture) or the operators 
(Heisenbcrg picture). In fact, it is the transformation 
of operators that provides the correct interpretation of 
the dynamics: physical properties at different times are 
related to each other in a uniquely defined state indepen- 
dent way. The relation between observations of the same 
physical property at different times is therefore equiv- 
alent to the relation between observations of different 
physical properties at the same time. In this sense, Lapla- 
cian determinism is also valid in quantum mechanics, and 
the deterministic relation of (a, b) and m expressed by the 
complex conditional probability p(m\a, b) is equally valid 
for the relation between initial position and momentum, 
and position at any later time. 

If Xt is the position of a particle at time t and (xq,pq) 
is the position and the momentum at time zero, the mo- 
tion of the particle is described by a complex conditional 
probability p(xt\xo,p$) that corresponds to the time de- 
pendent position operator x(t) of the Heisenberg picture, 

x(t) = J x t p(x t \x 0l po) | Po)(po I x ){x | dx t . (9) 

The complex conditional probability p(x t \xo,po) shows 
that the time evolution of x is still determined by the 
initial momentum, even though this dependence does not 
correspond to the linear evolution of classical physics. In- 
stead, the complex phase of the probability changes with 
increasing rates as Xt moves away from the classically 
predicted solution 5]. If the position of the object is 
observed with limited resolution, the rapid oscillations 
of the complex probabilities average out, leaving only 
results close to the classical solution for x t . Thus, the 
classical result emerges from complex conditional proba- 
bilities between the possible positions of the object, and 



not from a continuous movement of the center of mass 
along a mathematical line. Strictly speaking, the motion 
of objects observed in everyday life is therefore always 
associated with a complex probability distribution at the 
microscopic level, which is a necessary requirement of all 
motion observable at the macroscopic level. The famil- 
iar experiences of classical motion always involve interac- 
tions that allow us to observe the motion and therefore 
add sufficient noise to momentum and position to en- 
sure that all observations can be described by the coarse 
grained versions of p(x(t)\xo,Po)- As shown in such 
coarse graining quickly recovers the expectations of clas- 
sical determinism, explaining how the classical descrip- 
tion of motion emerges as a crude approximation of the 
quantum laws of motion expressed by the more precise 
complex conditional probabilities. 

Importantly, complex probabilities make a construc- 
tive contribution to our understanding of quantum dy- 
namics in the form of their imaginary parts. As discussed 
in the complex phase of the conditional probability 
p(m\a, b) describes the action required to optimally trans- 
form a to b along m, and this is the reason for the signif- 
icance of the constant h, which actually defines the ratio 
between the classical units of action and the phases of the 
complex conditional probabilities from which the action 
emerges in the approximate limit of classical physics. In 
the operator formalism, this relation appears in the well 
known Heisenberg equations of motion for the operators, 

|i=-i[i,£]. do 

This equation is often discussed in terms of the corre- 
spondence between commutation relations and the Pois- 
son brackets of classical physics. However, Eq.((3]) shows 
that the commutation relations between A and H also 
represent an imaginary correlation between the observ- 
able A and the energy H. It is therefore possible to find 
a macroscopic classical meaning in the imaginary corre- 
lations described by commutation relations. Specifically, 
the observable change of a property A in time is always 
equal to an imaginary correlation of this property and 
the energy, as given by the complex joint probability that 
characterizes the prior effects of the system, 

= ^^2A a E n Jm{p(n,a\ij)). (II) 

a,n 

Importantly, this relation only refers to the expectation 
value of A, which can be interpreted as an estimate of 
the actual value A a based on the prior information ip. 
The precision of the estimate is given by the uncertainty 
of A in ip. In the classical limit, this precision is usually 
much higher than the precision of observations of A, so 
the observed dynamics of A corresponds directly to the 
imaginary correlation. Eq. (fTT|) thus shows that classical 
motion invariably corresponds to microscopic imaginary 
correlations between the variable and the energy. 
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It is also possible to show that the equations of mo- 
tion require imaginary correlations between observations 
of the same property at two different times. In particu- 
lar, the velocity of an object of mass m is given by p/m, 
and the commutation relation of the momentum p and 
the position x is constant. Therefore, the two time cor- 
relation of position for an object freely moving through 
space has an imaginary part given by 



Im«i(t a )£(t 1 ))) = — (fc-ti). (12) 



Interestingly, this imaginary correlation increases in time, 
so that it would eventually become noticeable on a 
macroscopic level. However, quantum mechanics requires 
that an object can only move freely if there are no mea- 
surement interactions, and hence no measurements of the 
position between t\ and £2- If there is any interaction, 
it will reduce the correlations between the positions, just 
as it reduces the classical correlation between initial mo- 
mentum and final position. It is therefore possible to 
derive the magnitude of the disturbances necessary for a 
continuous observation of motion from the evolution of 
imaginary correlations given by Eq. (|12p . 



V. CONCLUSIONS 

The results observed in weak measurements and the 
definition of measurement uncertainties introduced by 
Ozawa both indicate that the quantum formalism can 
provide an uncertainty free description of physical real- 
ity. Here, I argue that complex conditional probabilities 
are the adequate expression for the fundamental relations 
between the different physical properties of a system. In 
this context, a physical property should be understood as 
a possible effect of the system, so that the assignment of 
a specific value or outcome depends on the presence of an 
interaction that produces the corresponding effect. Sig- 
nificantly, the complex conditional probabilities observed 
in weak measurements provide a context independent de- 
scription of the fundamental relation between pairs of 
properties (a, b) and any third property m, just as classi- 
cal mechanics uniquely determines the relation between 
position and momentum at time zero and the position at 
any other time. Complex probabilities can thus provide 
a more detailed microscopic explanation of quantum me- 
chanics, where the relation between different properties 
is fundamentally related to the action of transformations 
between the properties. 
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